Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus comprising: background generation means for generating, on each of corresponding pixel positions of more than one input image, a pixel value involved in a predetermined distribution among pixel values of the input images as a background pixel value in the pixel position; degree-of-difference generation means for generating, on each of the pixel positions, a degree-of-difference indicating a degree to which each input image pixel value has a difference from the background pixel value; and output image generation means for generating, on each of the pixel positions, an output image pixel value by reflecting the input image pixel values depending on the degree-of-difference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, moreparticularly, an image processing apparatus that displays a locusinvolved in an image provided by being imaged in time series, an imageprocessing method thereof, and a program adapted for a computer toperform the method.

2. Description of Related Art

Simple synthesis of more than one image (which is hereinafter referredto as a frame image) by overlapping, for instance, is suggested as onemethod for generating an image indicative of a moving object locus frommore than one frame image provided by being imaged in time series.According to the simple synthesis of this type, a rate of single framecontribution to a synthesis result decreases as the number of frames isincreased, causing a problem that the moving object locus displayedbecomes light-colored. When assuming that there are five frame images asshown in FIG. 10, for instance, the synthesis simply made on these frameimages causes the rate of each pixel contribution to the synthesisresult to decrease down to 20%. And, in a pixel position at which amoving object traverses, the synthesis is provided in a movingobject-to-background ratio of 20:80, leading to an occurrence oflight-colored display of the moving object locus as shown in FIG. 1C.

On the other hand, one technique provided by making improvements on thesimple synthesis is suggested to synthesize a single still image by,after extracting more than one frame image from video data, givingweighting suited for a luminance value (See Japanese Patent ApplicationPublication Hei. 10-290450 (FIG. 1), for instance). The techniqueaccording to this related art provides the synthesis on same positionalpixels involved in each frame image by heavily weighting the pixels thatare away from a mean luminance value by a distance or more of adispersion of the luminance values, while lightly weighting the pixelsthat are away from the mean luminance value by the distance or less ofthe above dispersion, followed by output of the synthesis result as thesame positional pixels of an output image. This technique ensures thatin a synthesized image, the moving object is more heavily weighted thana background, causing the moving object locus to be displayed moreclearly, as compared with the simple synthesis.

In addition, one different technique is suggested to provide thesynthesis of the moving object locus by extracting and tracing themoving object (See Japanese Patent Application Publication 2005-123824(Refer to FIG. 1), for instance). The technique according to thisrelated art ensures that an enhanced accuracy of extraction and tracingof the moving object is obtained by, when the tracing of the movingobject results in a failure, interpolating the moving object based on amoving object characteristic amount and/or position provided at the timeof successful tracing.

SUMMARY OF THE INVENTION

The synthesis with the moving object heavily weighted, like the methodaccording to the above-described related arts, has an effect ofpermitting the moving object locus to be displayed more clearly, ascompared with the simple synthesis. However, equal weighting isattributed to multiple synthesis except in the background, so that therate of single frame contribution to the locus decreases as the numberof frame images other than the background is increased, in which case,it is difficult to create the locus as clearly displayed as an originalimage. In addition, a method for judging the background based on thedispersion of the luminance values sometimes has difficulty in makingclear distinctions between the background and the other depending on acondition of the image.

Further, the technique according to the related art of a type thatinvolves the tracing of the moving object permits the locus to bedisplayed in an arbitrary fashion, while making interpolations of themoving object. However, in this case, moving object characteristics suchas region of existence, color, luminance and texture are used as thecharacteristic amount, so that a problem of being inadaptable for achange and/or a transformation of these characteristics is caused.Further, a presence of a large number of moving objects gives rise to aproblem of getting into difficulty in tracing all the moving objectsfrom a viewpoint of a cost taken for an arithmetic operation etc.

The present invention is intended to generate a locus-synthesized imageof a moving object from more than one frame image provided by beingimaged in time series, without being affected by a condition of abackground in the frame image and/or moving object characteristics suchas number, size, transformation and color.

The present invention has been undertaken in order to solve the aboveproblems, and its first aspect is to provide an image processingapparatus having background generation means; degree-of-differencegeneration means; and output image generation means. The backgroundgeneration means generates, on each of corresponding pixel positions ofmore than one input image, a pixel value involved in a predetermineddistribution among pixel values of the input images as a backgroundpixel value in the above pixel position. The degree-of-differencegeneration means generates, on each of the above pixel positions, adegree-of-difference indicating a degree to which each input image pixelvalue has a difference from the background pixel value. The output imagegeneration means generates, on each of the above pixel positions, anoutput image pixel value by reflecting the input image pixel valuesdepending on the degree-of-difference. This configuration produces aneffect of permitting the output image pixel value to be generateddepending on the degree-of-difference between each input image pixelvalue and the background pixel value.

Also, in the first aspect of the present invention, the backgroundgeneration means may be to generate a most frequently occurring pixelvalue as the above pixel value involved in the predetermineddistribution. Consideration is given to use of a movingobject-to-background difference in that the moving object has acharacteristic feature of making a transitory passing, while thebackground relatively has consistency.

Further, in the first aspect of the present invention, the backgroundgeneration means may also have background pixel value holding means;frequency-of-occurrence holding means; reference value setting means;background candidate generation means; and update determination means.The background pixel value holding means holds the background pixelvalue. The frequency-of-occurrence holding means holds afrequency-of-occurrence of the background pixel value. The referencevalue setting means sets a reference value for the input image pixelvalues. The background candidate generation means determines, as abackground candidate value, the input image pixel value that falls in apredetermined range of difference from the reference value. The updatedetermination means updates, after counting the frequency-of-occurrenceof the background candidate value, so as to cause thefrequency-of-occurrence of the background candidate value to be held inthe frequency-of-occurrence holding means, and also, the backgroundcandidate value to be held as a latest background pixel value in thebackground pixel value holding means when the frequency-of-occurrence ofthe background candidate value is greater than thefrequency-of-occurrence held in the frequency-of-occurrence holdingmeans. This configuration produces an effect of permitting the mostfrequently occurring pixel value to be held in the background pixelvalue holding means, without holding all the frequencies-of-occurrenceof the respective pixel values.

Furthermore, in the first aspect of the present invention, the outputimage generation means may also have output image holding means;synthesis ratio generation means; and synthesis value calculation means.The output image holding means holds the output image pixel value. Thesynthesis ratio generation means generates, on each of the above pixelpositions, a synthesis ratio depending on the degree-of-difference. Thesynthesis value calculation means synthesizes, on each of the abovepixel positions, the input image pixel values with the output imagepixel value held in the output image holding means depending on thesynthesis ratio, causing a synthesis result to be held as a latestoutput image pixel value in the output image holding means. Thisconfiguration produces an effect of permitting the output image pixelvalue to be generated depending on the generated synthesis ratio fromthe degree-of-difference.

Furthermore, in the first aspect of the present invention, the outputimage generation means may also have output image holding means; framenumber holding means; and update determination means. The output imageholding means holds the output image pixel value. The frame numberholding means holds an output image frame number. The updatedetermination means causes the input image frame number to be held inthe frame number holding means, and also, the input image pixel value tobe held as a latest output image pixel value in the output image holdingmeans when the input image frame number is more up-to-date in timeseries than the frame number held in the frame number holding means,provided that the degree-of-difference given is not less than apredetermined degree. This configuration produces an effect ofpermitting only the pixel value of the input image corresponding to alatest frame number to be reflected in the output image.

Furthermore, in the first aspect of the present invention, the outputimage generation means may also have output image holding means;degree-of-difference holding means; and update determination means. Theoutput image holding means holds the output image pixel value. Thedegree-of-difference holding means holds a degree-of-difference on theoutput image. The update determination means causes adegree-of-difference on the input image to be held in thedegree-of-difference holding means, and also, the input image pixelvalue to be held as a latest output image pixel value in the outputimage holding means when the degree-of-difference on the input image isgreater than the degree-of-difference held in the degree-of-differenceholding means. This configuration produces an effect of permitting onlythe pixel value of the input image having a maximum degree-of-differenceto be reflected in the output image.

A second aspect of the present invention is to provide an imageprocessing apparatus having image selection means; background generationmeans; degree-of-difference generation means; and output imagegeneration means. The image selection means selects a predeterminednumber of input images as selected images from more than one inputimage. The background generation means generates, on each ofcorresponding pixel positions of the selected images, a pixel valueinvolved in a predetermined distribution among pixel values of theselected images as a background pixel value in the above pixel position.The degree-of-difference generation means generates, on each of theabove pixel positions, a degree-of-difference indicating a degree towhich each input image pixel value has a difference from the backgroundpixel value. The output image generation means generates, on each of theabove pixel positions, an output image pixel value by reflecting theinput image pixel values depending on the degree-of-difference. Thisconfiguration produces an effect of permitting the number of selectedimages used to generate the background pixel value to be setarbitrarily.

A third aspect of the present invention is to provide an imageprocessing apparatus having background image holding means; output imageholding means; background generation means; degree-of-differencegeneration means; and output image synthesis means. The background imageholding means holds a background image pixel value. The output imageholding means holds an output image pixel value. The backgroundgeneration means updates the background image pixel value held in thebackground image holding means based on a predetermined ratio and withpixel values in corresponding pixel positions of input images allowed toenter in time series. The degree-of-difference generation meansgenerates a degree-of-difference indicating a degree to which each inputimage pixel value has a difference from the pixel value in thecorresponding pixel position of the updated background image provided bythe background generation means. The output image synthesis meanssynthesizes, on the output image pixel value, the pixel values in thecorresponding pixel positions of the input images depending on thedegree-of-difference, causing a synthesis result to be held as a latestoutput image pixel value in the output image holding means. Thisconfiguration produces an effect of permitting the output image pixelvalue to be generated depending on the degree-of-difference from eachinput image pixel value by updating the background image with the inputimages allowed to enter in time series.

A fourth aspect of the present invention is to provide an imageprocessing apparatus having image selection means; background generationmeans; background image holding means; output image holding means;degree-of-difference generation means; and output image synthesis means.The image selection means selects a predetermined number of input imagesas selected images from more than one input image. The backgroundgeneration means generates, on each of corresponding pixel positions ofthe selected images, a pixel value involved in a predetermineddistribution among pixel values of the selected images as a pixel valuein a corresponding pixel position of a background image. The backgroundimage holding means holds the background image pixel value. The outputimage holding means holds an output image pixel value. Thedegree-of-difference generation means generates a degree-of-differenceindicating a degree to which each input image pixel value has adifference from the corresponding background image pixel value held inthe background image holding means. The output image synthesis meanssynthesizes, on the output image pixel value, the pixel values in thecorresponding pixel positions of the input images depending on thedegree-of-difference, causing a synthesis result to be held as a latestoutput image pixel value in the output image holding means. Thisconfiguration produces an effect of permitting the number of selectedimages used to generate the background pixel value to be setarbitrarily.

A fifth aspect of the present invention is to provide an imageprocessing apparatus having image selection means; background generationmeans; background image holding means; output image holding means;degree-of-difference generation means; output image synthesis means;background image storage means. The image selection means selects apredetermined number of input images as selected images from more thanone input image. The background generation means generates, on each ofcorresponding pixel positions of the selected images, a pixel valueinvolved in a predetermined distribution among pixel values of theselected images as a pixel value in a corresponding pixel position of abackground image. The background image holding means holds thebackground image pixel value. The output image holding means holds anoutput image pixel value. The degree-of-difference generation meansgenerates a degree-of-difference indicating a degree to which each inputimage pixel value has a difference from the corresponding backgroundimage pixel value held in the background image holding means. The outputimage synthesis means synthesizes, on the output image pixel value, thepixel values in the corresponding pixel positions of the input imagesdepending on the degree-of-difference, causing a synthesis result to beheld as a latest output image pixel value in the output image holdingmeans. The background image storage means for storing the backgroundimage pixel value held in the background image holding means. Thebackground image holding means makes restoration of the background imagestored in the background image storage means before holding, when arequired background image is contained in the background image storagemeans. This configuration produces an effect of permitting processingfor generating the background image to be saved by using the backgroundimage in the back stored ground image storage means through therestoration.

According to the present invention, the effects may be obtained asexcellent as of generating the locus-synthesized image of the movingobject from more than one frame-image provided by being imaged in timeseries, without being affected by the condition of the background in theframe image and/or the moving object characteristics such as the number,the size, the deformation and the color.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing one instance of an imaging apparatusaccording to one embodiment of the present invention:

FIG. 2 is a block diagram showing a first example of an image processingcircuit of the imaging apparatus according to the embodiment of thepresent invention;

FIG. 3 is a block diagram showing one instance of a first configurationof a background generation unit included in the first example of theimage processing circuit according to one embodiment of the presentinvention;

FIG. 4 is a graphic representation example of one distribution of pixelvalues;

FIG. 5 is a graphic representation of one instance of a histogram basedon the distribution of the pixel values;

FIG. 6 is a block diagram showing one instance of a second configurationof the background generation unit included in the first example of theimage processing circuit according to the embodiment of the presentinvention;

FIG. 7 is a graphic representation of one instance of a reference valueand a background threshold according to the embodiment of the presentinvention;

FIG. 8 is a block diagram showing one configuration of adegree-of-difference generation unit included in the first example ofthe image processing circuit according to the embodiment of the presentinvention;

FIG. 9 is a block diagram showing one instance of a first configurationof a locus synthesis unit included in the first example of the imageprocessing circuit according to the embodiment of the present invention;

FIG. 10 shows one instance of an input image;

FIG. 11 shows one instance of a locus-synthesized image;

FIG. 12 is a block diagram showing one instance of a secondconfiguration of the locus synthesis unit included in the first exampleof the image processing circuit according to the embodiment of thepresent invention;

FIG. 13 is a block diagram showing one instance of a third configurationof the locus synthesis unit included in the first example of the imageprocessing circuit according to the embodiment of the present invention;

FIG. 14 is a block diagram showing a second example of the imageprocessing circuit of the imaging apparatus according to one embodimentof the present invention;

FIG. 15 is a block diagram showing a third example of the imageprocessing circuit of the imaging apparatus according to one embodimentof the present invention;

FIG. 16 is a block diagram showing one configuration of a backgroundgeneration unit included in the third example of the image processingcircuit according to the embodiment of the present invention;

FIG. 17 is a block diagram showing one configuration of a locussynthesis unit included in the third example of the image processingcircuit according to the embodiment of the present invention;

FIG. 18 shows one instance of each frame involved in a locus-synthesizedmoving image;

FIG. 19 shows another instance of each frame involved in thelocus-synthesized moving image;

FIG. 20 is a block diagram showing a fourth example of the imageprocessing circuit of the imaging apparatus according to the embodimentof the present invention;

FIG. 21 is a flowchart showing one procedure of processing with thefirst and the second examples of the image processing circuit of theimaging apparatus according to the embodiment of the present invention;

FIG. 22 is a flowchart showing one procedure of processing with thethird example of the image processing circuit of the imaging apparatusaccording to the embodiment of the present invention;

FIG. 23 is a flowchart showing one procedure of processing with thefourth example of the image processing circuit of the imaging apparatusaccording to the embodiment of the present invention;

FIG. 24 is a block diagram showing a modification of the imageprocessing circuit of the imaging apparatus according to the embodimentof the present invention;

FIG. 25 is a block diagram showing one configuration related tooperations of the imaging apparatus according to the embodiment of thepresent invention;

FIG. 26 is a flowchart showing one procedure of processing wheneffecting an image-taking operation with the imaging apparatus accordingto one embodiment of the present invention; and

FIG. 27 is a flowchart showing one procedure of processing wheneffecting a playback operation with the imaging apparatus according toone embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

FIG. 1 is a block diagram showing one instance of an imaging apparatusaccording to one embodiment of the present invention. Roughly speaking,the imaging apparatus shown is composed of an optical system, a signalprocessing system, a recording system, a display system and a controlsystem.

The optical system is composed of a lens 11 that focuses an opticalimage of an object, a diaphragm 12 that controls an amount of light ofthe optical image, and an imaging device 13 that photoelectric-convertsthe focused optical image into an electric signal. The imaging device 13is implemented with an image sensor such as CCD (Charge CoupledDevices), for instance.

The signal processing system is composed of a sampling circuit 21 thateffects sampling of the electric signal supplied from the imaging device13, an A/D conversion circuit 22 that converts an analog signal suppliedfrom the sampling circuit 21 into a digital signal, and an imageprocessing circuit 23 that gives predetermined image processing to thedigital signal supplied from the A/D conversion circuit 22. The samplingcircuit 21 is implemented with a correlated double sampling circuit(CDS), for instance. Use of the CDS circuit may get relief from noisesoriginating in the imaging device 13. It is to be noted that processingto be performed with the image processing circuit 23 is detailed later.

The recording system is composed of a memory 32 used to store an imagesignal, and an encoder/decoder 31 that encodes and records, in thememory 32, the image signal given the processing by the image processingcircuit 23, and also, after reading out the image signal from the memory32, decodes and supplies the read image signal to the image processingcircuit 23.

The display system is composed of a D/A conversion circuit 41 thatanalogizes the image signal given the processing by the image processingcircuit 23, a video encoder 42 that encodes the analogized image signalinto a video signal of a form adapted for a post-staged display unit 43,and the display unit 43 that displays an image corresponding to thevideo signal inputted. The display unit 43 is implemented with a LCD(Liquid Crystal display) etc., for instance, and also provides afunction as a viewfinder.

The control system is composed of a timing generator 51 that controlsoperation timings of the imaging device 13, the sampling circuit 21, theA/D conversion circuit 22 and the image processing circuit 23, anoperation input acceptance unit 52 used to accept input of a shutteroperation and other commands given by a user, a driver 53 used to makeconnection with a peripheral equipment, and a control unit 54 thatcontrols the imaging apparatus in its entirety. The driver 53 isconnected to peripheral equipment such as a magnetic disk, an opticaldisc, a magneto-optical disc and a semiconductor memory. The controlunit 54 reads out, through the driver 53, a control program placed inthe above peripheral equipment, and then provides control based on theread control program and/or the commands etc. given by the user throughthe operation input acceptance unit 52.

The image processing circuit 23, the encoder/decoder 31, the memory 32,the timing generator 51, the operation input acceptance unit 52 and thecontrol unit 54 are interconnected through a bus 59.

As for the imaging apparatus of this type, the optical image (incidentlight) of the object is allowed to enter the imaging device 13 throughthe lens 11 and the diaphragm 12, and is then photoelectric-convertedinto the electric signal by the imaging device 13. The electric signalobtained undergoes removal of noise components by the sampling circuit21, and is then digitized by the A/D conversion circuit 22, followed bybeing temporarily stored in an image memory (not shown) integrated inthe image processing circuit 23.

It is to be noted that in a normal condition, control by the timinggenerator 51 to the signal processing system is adapted to providecontinual overwriting of the image signal at a certain frame rate forthe image memory integrated in the image processing circuit 23. Theimage signal in the image memory integrated in the image processingcircuit 23 is converted into the analog signal by the D/A conversioncircuit 41, and further, into the video signal by the video encoder 42,causing a corresponding image to be displayed on the display unit 43.

The display unit 43 also functions as the viewfinder of the imagingapparatus. If a shutter button included in the operation inputacceptance unit 52 is pushed down by the user, the control unit 54permits the timing generator 51 to control the signal processing systemso as to hold the image signal provided immediately after the shutterbutton is pushed down, or to prevent the image signal from beingoverwritten in the image memory of the image processing circuit 23.Image data placed in the image memory of the image processing circuit 23is encoded by the encoder/decoder 31, and is then recorded in the memory32. Operations of the imaging apparatus as described the above causecapturing of a single frame of image data to be completed.

FIG. 2 is a block diagram showing a first example of the imageprocessing circuit 23 of the imaging apparatus according to oneembodiment of the present invention. The image processing circuit 23according to the first. embodiment has a pixel selection unit 120, abackground generation unit 130, a degree-of-difference generation unit140, a locus synthesis unit 150, and a locus-synthesized image displayunit 190.

The pixel selection unit 120 is to select n (n is an integer of not lessthan 2) pieces of pixel values in corresponding pixel positions as for nframes of input images 110 (I1 to In) provided by being imaged in timeseries. Assuming that each of the input images 110 is composed of p×qpixels (p and q are integers of not less than 1) in coordinates from(1,1) to (p,q), for instance, the pixel selection unit 120 firstlyselects each pixel value in the pixel position of the coordinate (1,1)out of the n frames of input images 110, causing the n pieces of pixelvalues to be selected. Subsequently, the pixel selection unit 120selects each pixel value in the pixel position of the coordinate (1,2)out of the n frames of input images 110, causing the following n piecesof pixel values to be selected. Finally, the n pieces of pixel valuesare selected in this manner as many as for p×q groups of the pixels withrespect to the input images 110 (I1 to In). These selected pixel valuesare supplied to the background generation unit 130, thedegree-of-difference generation unit 140 and the locus synthesis unit150 through a signal line 129.

The background generation unit 130 is to generate a background valuebased on the n pieces of pixel values supplied from the pixel selectionunit 120 through the signal line 129. The n pieces of pixel values aresupplied as many as for the p×q groups of the pixels as described theabove, so that the background generation unit 130 finally yields p×qpieces of background values with respect to the input images 110 (I1 toIn). These background values are supplied to the degree-of-differencegeneration unit 140 through a signal line 139.

The degree-of-difference generation unit 140 is to generate adegree-of-difference between the background value supplied from thebackground generation unit 130 through the signal line 139 and eachpixel value supplied from the pixel selection unit 120 through thesignal line 129. As described the above, each background value composedof p×q pixels is generated by the background generation unit 130. Thus,the degree-of-difference generation unit 140 generates respectively thedegree-of-difference between the background value generated and each ofthe n pieces of pixel values of the input images 110 (I1 to In) for eachbackground value pixel position, finally causing the n pieces ofdegrees-of-difference to be generated as many as for the p×q groups ofthe pixels with respect to the input images 110 (I1 to In). Thesegenerated degrees-of-difference are supplied to the locus synthesis unit150 through a signal line 149.

The locus synthesis unit 150 is to synthesize the pixel values suppliedfrom the pixel selection unit 120 through the signal line 129 based onthe degree-of-difference supplied from the degree-of-differencegeneration unit 140 through the signal line 149. As described the above,the n pieces of degrees-of-difference are generated as many as for thep×q groups of the pixels, and the n pieces of pixel values are selectedas many as for the p×q groups of the pixels. In other words, thedegrees-of-difference agree with the pixel values in number. The locussynthesis unit 150 finally yields a single pixel value as a locussynthesis value from the n pieces of pixel values, causing alocus-synthesized image having p×q pieces of locus synthesis. values tobe generated. These locus synthesis values are supplied to thelocus-synthesized image display unit 190 through a signal line 159.

The locus-synthesized image display unit 190 is to display thelocus-synthesized image based on the locus synthesis values suppliedfrom the locus synthesis unit 150 through the signal line 159. Thelocus-synthesized image display unit 190 may be shared with the displayunit 43 intact, or alternatively, may be implemented with a differentdisplay apparatus connected to the imaging apparatus.

FIG. 3 shows one instance of a first configuration example of thebackground generation unit 130 included in the first example of theimage processing circuit 23 according to the embodiment of the presentinvention. The background generation unit 130 shown has a histogramgeneration unit 131 and a background value determination unit 132.

The histogram generation unit 131 is to generate a histogram on the npieces of pixel values supplied from the pixel selection unit 120through the signal line 129. The histogram generated expresses afrequency-of-occurrence of the pixel values as against their sizes.Assuming that a distribution of the pixel values exists as shown in FIG.4, for instance, the histogram on these pixel values is presented in theform of a histogram as shown in FIG. 5. Specifically, for thefrequencies-of-occurrence on 13 pieces of pixel values in FIG. 4, pixelvalues a, b, c, d and e respectively occur twice, once, seven times,once and twice, in which case, the histogram as shown in FIG. 5 isgenerated.

It is to be noted that the histogram is presented in the form of aone-dimensional histogram as shown in FIG. 4 if luminance is only takeninto account, or may be made in the form of a three-dimensionalhistogram if RGB-type pixel values are provided. In other words, thehistogram is supposed to be generated in the form of any dimensionalhistogram suited to a color space to be defined.

The background value determination unit 132 is to determine thebackground value depending on the histogram generated in the histogramgeneration unit 131. Specifically, the background value determinationunit 132 determines a most frequently occurring pixel value in thehistogram to be the background value. In case as shown in FIG. 4, forinstance, the pixel value c offering a mode is determined to be thebackground value.

FIG. 6 shows one instance of a second configuration of the backgroundgeneration unit 130 included in the first example of the imageprocessing circuit 23 according to the embodiment of the presentinvention. The background generation unit 130 shown has a backgroundthreshold setting unit 101, a reference value setting unit 133, areference value counter 1331, a background determination unit 134, abackground value calculation unit 135, an update determination unit 136,a background frequency counter 1361, a background frequency holding-unit137 and a background value holding unit 138.

The background determination unit 134 is to determine whether or not thepixel values of the input images 110 are of the background. Thereference value setting unit 133 is to set a value (or a referencevalue) serving as a reference used for the background determination unit134 to determine the background. The reference value counter 1331 is tosequentially supply the reference value that is to be set in thereference value setting unit 133. The background threshold setting unit101 is to supply a threshold (or a background threshold) yielded fromthe reference value used for the background determination unit 134 todetermine the background. The background determination unit 134calculates a distance between the reference value set in the referencevalue setting unit 133 and each pixel value of the input images 110, andthen determines the pixel value distant from the reference value by thebackground threshold or less to be of the background. It is to be notedthat the distance in this case involves a color spatial distance etc.,for instance. Thus, in a case of a multi-dimensional space like an RGBspace, for instance, the distance from the reference value by thebackground threshold or less respectively in all dimensions is judged tobe of the background threshold or less.

For the distribution of the pixel values as shown in FIG. 7, if thereference value is given as illustrated, for instance, a pixel valuerange including upper and lower background thresholds respectively givento the pixel values centering around the given reference value isregarded as a background range. The reference value to be set in thereference value setting unit 133 undergoes sequential changes by thereference value counter 1331. The reference value is to be sequentiallyset in ascending or descending order of the pixel values, for instance.This processing causes determination whether or not each pixel value isin the range of the background as seen from the reference value to bemade by the background determination unit 134 over the whole range ofthe pixel values.

The background value calculation unit 135 calculates the backgroundvalue from the pixel values determined to be of the background by thebackground determination unit 134. The background value is calculated asa mean of the pixel values determined to be of the background by thebackground determination unit 134, for instance.

The update determination unit 136 is to, after counting with thebackground frequency counter 1361, the frequency of the pixel valuesdetermined to be of the background by the background determination unit134, determine whether or not updating should be made to the backgroundfrequency holding unit 137 and the background value holding unit 138.Specifically, only in a case where a count value provided by thebackground frequency counter 1361 is greater than a frequency value heldin the background frequency holding unit 137, the updating is made so asto replace contents in the background frequency holding unit 137 withthe count value provided by the background frequency counter 1361,together with replacement of contents in the background value holdingunit 138 with the background value supplied from the background valuecalculation unit 135. This processing causes a mode value of the pixelvalue at that time to be held in the background value holding unit 138,and also, the frequency of the mode pixel value to be held in thebackground frequency holding unit 137. The sequential changes to thereference value setting unit 133 so as to set all the pixel values asthe reference value finally causes the pixel value offering the mode tobe held in the background value holding unit 138.

While the first configuration example of the background generation unit130 in FIG. 3 requires a memory area to store the histogram in itsentirety, the second configuration example of the background generationunit 130 in FIG. 6 is satisfied only by holding the frequency of thepixel value offering the mode at that time, and thus, enables memoryarea saving.

FIG. 8 is a block diagram showing one configuration example of thedegree-of-difference generation unit 140 included in the first exampleof the image processing circuit 23 according to the embodiment of thepresent invention. The degree-of-difference generation unit 140 shownhas a distance calculation unit 141, a distance threshold setting unit102, and a degree-of-difference calculation unit 142.

The distance calculation unit 141 is to calculate the distance betweenthe background value supplied from the background generation unit 130through the signal line 139 and each pixel value supplied from the pixelselection unit 120 through the signal line 129. It is to be noted thatthe distance in this case involves the color spatial distance etc., forinstance.

The degree-of-difference calculation unit 142 is to calculate thedegree-of-difference between the background value and each input imagepixel value depending on the distance calculated by the distancecalculation unit 141. Specifically, the degree-of-difference calculationunit 142 is to distinguish in each input image between a moving objectand the background, providing a moving object-to-background differenceas the degree-of-difference. While the degree-of-difference is to besettled so as to assign “1” to the moving object, and “0” to thebackground, any intermediate value between “1” and “0” may be sometimestaken if the moving object and the background are not completelydistinguishable from each other.

In calculating the degree-of-difference, the degree-of-differencecalculation unit 142 assumes use of a threshold (or a distancethreshold) of the distance to distinguish between the background and themoving object in considerations of an error depending on an imagequality. Specifically, when the distance between the background valueand each input image pixel value is within the distance threshold, thebackground and the moving object are treated as being not completelydistinguishable from each other, while when exceeding the distancethreshold, the background and the moving object are treated as beingcompletely distinguishable from each other. As a result, in a case thedistance between the background value and each input image pixel valueexceeds the distance threshold, “1” specified as a maximum degree ofdifference is settled, while in a case of being within the distancethreshold, the degree-of-difference is settled in proportion to thedistance, for instance. It is to be noted that the distance threshold isto be preliminarily set in the distance threshold setting unit 102.

FIG. 9 is a block diagram showing one instance of a first configurationof the locus synthesis unit 150 included in the first example of theimage processing circuit 23 according to one embodiment of the presentinvention. The locus synthesis unit 150 shown has a locus synthesisratio generation unit 151, a locus synthesis value calculation unit 152and a locus synthesis value holding unit 153.

The locus synthesis ratio generation unit 151 is to generate a ratio (ora locus synthesis ratio) for synthesis of the input image pixel valuesas a locus based on the degree-of-difference supplied from thedegree-of-difference generation unit 140 through the signal line 149.The locus synthesis ratio generated may be settled in proportion to thedegree-of-difference, for instance, or alternatively, by taking accountof the degree-of-difference inclusive of that around the input imagepixels.

The locus synthesis value holding unit 153 is to hold an output imagepixel value (or a locus synthesis value) provided from the locussynthesis unit 150. The locus synthesis value calculation unit 152 is tosynthesize, depending on the locus synthesis ratio generated by thelocus synthesis ratio generation unit 151, the input image pixel valueswith respect to the pixel values held in the locus synthesis valueholding unit 153. Specifically, the locus synthesis value to be held inthe locus synthesis value holding unit 153 is supposed to be updatedwith the locus synthesis value calculation unit 152 at any time.

Now assuming that the locus synthesis ratio is represented by β, thelocus synthesis value provided after updated based on a j-th input imageis represented by S_(j), a k(=j+1)-th input image is represented byI_(k), and the locus synthesis value provided after updated based on thek-th input image is represented by S_(k), the following expression isgiven.S _(k) =β×I _(k)+(1−β)×S _(j)Where j and k are integers, and β, Ik, Sj and Sk are real numbers.

Effects with the locus synthesis ratio β are now described by assumingthe images provided by being imaged in the order given in FIGS. 10A,10B, 10C, 10D and 10E to be the input images. In a case where the locussynthesis value β is assigned as “1” to a moving object area having agreat degree-of-difference and as “0” to a background area having asmall degree-of-difference, the background and the moving object aresynthesized without being merged, causing all moving object loci to bedisplayed dark-colored, as shown in FIG. 11A.

On the other hand, in a case where the locus synthesis value β isassigned as “1” to the moving object area, and as about “0.3” to “0.5”to the background area, the background gets merged with the movingobject every time the synthesis is provided, causing the moving objectloci to be displayed so as to become more light-colored as the locusbecomes older, as shown in FIG. 11B.

FIG. 12 is a block diagram showing a second configuration example of thelocus synthesis unit 150 included in the first example of the imageprocessing circuit 23 according to the embodiment of the presentinvention. The locus synthesis unit 150 shown has an updatedetermination unit 154, a latest frame number holding unit 155, and alocus synthesis value holding unit 156. The second configuration exampleof the locus synthesis unit 150 controls, in view of a case where inputin imaging order is not applied to the input images 110 (I1 to In), topermit only the latest pixel value in each pixel position to be updated.Specifically, when the ascending consecutive numbers are assigned asframe numbers respectively to the input images 110 in the ascendingorder, the pixel value assigned with the greater frame number is assumedto be a more up-to-date pixel value. It is to be noted that the framenumbers are to be supplied together with the pixel values through thesignal line 129.

The locus synthesis value holding unit 156 is to hold the locussynthesis value, like the locus synthesis value holding unit 153.However, the second configuration example specifies that the backgroundvalue supplied through the signal line 139 is to be held as an initialvalue in advance.

The latest frame number holding unit 155 is to hold a maximum among theframe numbers of the input images 110 having been allowed to enter tillthen in a current pixel position.

The update determination unit 154 compares each frame number suppliedthrough the signal line 129 with the frame number held in the latestframe number holding unit 155, and then causes the pixel value suppliedthrough the signal line 129 to be held in the locus synthesis valueholding unit 156 if the frame number supplied through the signal line129 is more up-to-date, provided that no relation to the backgroundexists. However, the update determination unit 154 allows the backgroundvalue supplied through the signal line 139 to be held as the initialvalue in the locus synthesis value holding unit 156, before the firstinput image in each pixel position is selected. This processing causesthe updating of only the latest and background-unrelated pixel value tobe provided for the locus synthesis value holding unit 156 so that themost up-to-date moving object is given as the final locus synthesisvalue, permitting the locus synthesis value in view of time series orderto be obtained.

FIG. 13 is a block diagram showing a third configuration example of thelocus synthesis unit 150 included in the first example of the imageprocessing circuit 23 according to one embodiment of the presentinvention. The locus synthesis unit 150 shown has an updatedetermination unit 157, a maximum degree-of-difference holding unit 158,and a locus synthesis value holding unit 156. The locus synthesis valueholding unit 156 is the same as that included in the secondconfiguration example of the locus synthesis unit 150.

The maximum degree-of-difference holding unit 158 is to hold a maximumdegree-of-difference having been allowed to enter till then in thecurrent pixel position.

The update determination unit 157 compares each degree-of-differencesupplied through the signal line 149 with the degree-of-difference heldin the maximum degree-of-difference holding unit 158, and then causesthe pixel value supplied through the signal line 129 to be held in thelocus synthesis value holding unit 156 when the degree-of-differencesupplied through the signal line 129 is greater. This processing causesthe updating of only the pixel value having the maximumdegree-of-difference to be provided for the locus synthesis valueholding unit 156, permitting the locus having the greaterdegree-of-difference to be reflected in an area containing overlapped.moving objects.

FIG. 14 is a block diagram showing a second embodiment of the imageprocessing circuit 23 of the imaging apparatus according to oneembodiment of the present invention. The image processing circuit 23according to the second embodiments has a locus pixel selection unit121, a background pixel selection unit 122, the background generationunit 130, the degree-of-difference generation unit 140, the locussynthesis unit 150 and the locus-synthesized image display unit 190.Specifically, the first embodiment is different from the secondembodiment in that the pixel selection unit 120 of the image processingcircuit 23 according to the first embodiment in FIG. 2 is separated intothe locus pixel selection unit 121 and the background pixel selectionunit 122 in the case of the second example.

The background pixel selection unit 122 is to select the required pixelvalue for generation of the background value out of the n pieces ofpixel values in the corresponding pixel positions as for the n frames ofinput images 110 (I1 to In) provided by being imaged in time series.While the first embodiment has made selection of all the n frames ofinput images to generate the background value, the second embodimentallows for simplification of processing for generating the backgroundvalue by using thinned-out images provided from the n frames of inputimages. For instance, thinning-out of the input images into “n/10” onthe pixel values selected by the background pixel selection unit 122enables an increase in processing speed.

The locus pixel selection unit 121 is to select the pixel value otherthan that of the background, or the pixel value of the moving objectlocus. The locus pixel selection unit 121 selects the n pieces of pixelvalues in the corresponding pixel positions as for the n frames of inputimages 110 (I1 to In), like the pixel selection unit 120 in the firstexample.

In this manner, the image processing circuit 23 in the second examplemay hold down a background calculation cost by thinning out the imageson the pixel values selected in the background pixel selection unit 122by taking advantage of a fact that a frequency-of-change of thebackground is lower, as compared with that of the moving object.

FIG. 15 is a block diagram showing a third example of the imageprocessing circuit 23 of the imaging apparatus according to oneembodiment of the present invention. The image processing circuit 23according to the third example has a frame selection unit 220, abackground generation unit 230, a background image holding unit 260, adegree-of-difference generation unit 240, a locus synthesis unit 250, alocus-synthesized image holding unit 270, and a locus-synthesized imagedisplay unit 290. While the first and the second examples are providedon the assumption that the moving object locus is displayed as thesingle locus-synthesized image, the third example assumes that anin-moving state of the moving object is to-be displayed as more than onelocus-synthesized image, as shown in FIGS. 18 and 19, for instance.

The frame selection unit 220 is to sequentially select an image I_(t)composed of p×q pixels as a frame from input images 111 in the timeseries order. The selected frame is sequentially supplied to thebackground generation unit 230, the degree-of-difference generation unit240 and the locus synthesis unit 250 through a signal line 229.

The background image holding unit 260 is to hold the background imagecomposed of the p×q pixels. The background image held in the backgroundimage holding unit 260 is supplied to the background generation unit 230through a signal line 269. The background generation unit 230 is togenerate a latest background image B_(t) based on the pixel value of abackground image B_(t−1) supplied from the background generation unit230 through the signal line 269 and the pixel value of the frame Itsupplied from the frame selection unit 220 through the. signal line 229.Specifically, the background generation unit 230 updates the backgroundimage to be held in the background image holding unit 260 by replacingthe image with a latest frame, causing the background image at that timeto be held in the background image holding unit 260. The generatedbackground image is supplied to the degree-of-difference generation unit240 and the background image holding unit 260 through a signal line 239.

The degree-of-difference generation unit 240 is to generate thedegree-of-difference on a pixel position basis between the pixel valueof the background image supplied from the background generation unit 230through the signal line 239 and the pixel value of the frame suppliedfrom the frame selection unit 220 through the signal line 229.Specifically, the degree-of-difference generation unit 240 generates thedegrees-of-difference as many as for the p×q groups of the pixels bycalculating the corresponding pixel value-to-pixel value distances asmany as for the p×q groups of the pixels. The degree-of-differencegeneration unit 240 has the same internal configuration as thedegree-of-difference generation unit 140 shown in FIG. 8.

The locus-synthesized image holding unit 270 is to hold thelocus-synthesized image consisting of the p×q pixels. Thelocus-synthesized image held in the locus-synthesized image holding unit270 is supplied to the locus synthesis unit 250 through a signal line279. The locus synthesis unit 250 is to generate a latestlocus-synthesized image S_(t) by synthesizing the pixel value of alocus-synthesized image S_(t−1) supplied from the locus-synthesizedimage holding unit 270 through the signal line 279 and the pixel valueof the frame I_(t) supplied from the frame selection unit 220 throughthe signal line 229. Specifically, the locus synthesis unit 250 updatesthe locus-synthesized image to be held in the locus-synthesized imageholding unit 270 by replacing the image with the latest frame, causingthe locus-synthesized image at that time to be held in thelocus-synthesized image holding unit 270.

The locus-synthesized image display unit 290 is to display thelocus-synthesized image S_(t) supplied from the locus synthesis unit 250through a signal line 259. The locus-synthesized image display unit 290may be shared with the display unit 43 intact, or alternatively, may beimplemented with the different apparatus connected to the imagingapparatus.

FIG. 16 is a block diagram showing one configuration example of thebackground generation unit 230 included in the third example of theimage processing circuit 23 according to one embodiment of the presentinvention. The background generation unit 230 shown has a backgroundsynthesis ratio generation unit 232 and a background image update unit233.

The background synthesis ratio generation unit 232 is to generate asynthesis ratio (or a background synthesis ratio a) of the backgroundimage based on the frame supplied from the frame selection unit 220through the signal line 229 and the background image supplied from thebackground image holding unit 260 through the signal line 269. Asspecific calculation of the background synthesis ratio a, one method issuggested, for instance, to determine the background synthesis ratio soas to provide a greater contribution for the pixel value of the frameI_(t) when the degree-of-difference between the pixel value of the frameI_(t) and the pixel value of the background image B_(t−1) is small, orto determine the synthesis ratio dynamically based on thedegree-of-difference so as to increase the contribution of the pixelvalue of the frame I_(t) when a small dispersion exists in temporallynearby degrees-of-difference. Alternatively, it is also allowable topreliminarily settle the background synthesis ratio a as a fixed value.

The background image update unit 233 is to update the background imagesupplied from the background image holding unit 260 through the signalline 269 by replacing the image with the frame supplied from the frameselection unit 220 through the signal line 229, depending on thebackground synthesis ratio a generated by the background synthesis ratiogeneration unit 232.

Now assuming that the pixel value of the frame I_(t) in a coordinate (x,y) is represented by I(x, y, t), and, likewise, the pixel value of thebackground image B_(t−1) is represented by (x, y, t−1), a pixel valueB(x, y, t) of the latest background image B_(t) is obtained by thefollowing expression.B(x, y, t)=α×I(x, y, t)+(1−α)×B (x, y, t−1)

The background image update unit 233 gives the processing meeting theabove expression to all the pixels.

While the above embodiment specifies that the single image I_(t) isselected as the frame by the frame selection unit 220, it may bemodified to supply the frame to the background generation unit 230 basedon more than one images. For instance, after finding, on the basis ofimages I_(t−i) to I_(t+i) arranged before and behind with the imageI_(t) between, a temporary background image from frequencies of theabove images, like the examples shown in FIG. 3 or 6, the temporarybackground image found may be supplied to the background generation unit230. This processing is expected to result in implementation of highlyaccurate background generation in the background generation unit 230.

FIG. 17 is a block diagram showing one configuration of the locussynthesis unit 250 included in the third example of the image processingcircuit 23 according to the embodiment of the present invention. Thelocus synthesis unit 250 shown has a locus synthesis ratio calculationunit 251 and a locus synthesis value update unit 252.

The locus synthesis ratio calculation unit 251 is to generate thesynthesis ratio (or the locus synthesis ratio β) for the locus,synthesis based on the degree-of-difference supplied from thedegree-of-difference generation unit 240 through the signal line 249. Asthe specific calculation of the locus synthesis ratio β, one method issuggested to determine the locus synthesis ratio so as to provide thegreater contribution for the pixel value of the frame I_(t) when thedegree-of-difference between the pixel value of the frame I_(t) and thepixel value of the background image B_(t−1) is great, for instance. Useof this method ensures that the rate of contribution to the locussynthesis value increases with the difference of the area from thebackground, permitting a more clearly displayed moving object locus tobe obtained.

The locus synthesis value update unit 252 is to update thelocus-synthesized image supplied from the locus-synthesized imageholding unit 270 through the signal line 279 by replacing the image withthe frame supplied from the frame selection unit 220 through the signalline 229, depending on the locus synthesis ratio β generated by thelocus synthesis ratio calculation unit 251.

Now assuming that the pixel value of the frame I_(t) in the coordinate(x, y) is represented by I(x, y, t), and likewise, the pixel value ofthe locus-synthesized image S_(t−1) is represented by S(x, y, t−1), apixel value S(x, y, t) of the latest locus-synthesized image St may beobtained by the following expression.S(x, y, t)=β×I(x, y, t)+(1−β)×S(x, y, t−1)

The locus-synthesized image update unit 252 gives the processing meetingthe above expression to all the pixels.

Alternatively, as a different-mannered locus synthesis, one method issuggested to calculate the locus synthesis ratio using two types ofcalculation rules based on the degree-of-difference. Specifically,provided that the locus synthesis ratio calculation rule related to thebackground area having the small degree-of-difference is used separatelyfrom the calculation rule for the moving object area having the greatdegree-of-difference, a certain degree of locus synthesis ratio is to begiven even if the degree-of-difference is small. Use of this methodensures that the rate of background contribution increases with theolder area of the moving object locus, permitting the moving objectlocus to be displayed in a fadeout form. That is, while the up-to-datelocus is clearly displayed, the other locus is allowed to fade out so asto be merged into the background with a passage of time.

In synthesizing the five frames as shown in FIG. 10, for instance, ifthe locus needs to leave as it is clearly displayed, assignment of thelocus synthesis ratio β as “1” to the moving object area having thegreat degree-of-difference, and as “0” to the background area having thesmall degree-of-difference may produce the result as shown in FIG. 18.On the other hand, if the locus needs to fade out, the assignment of thelocus synthesis ratio g as “1” to the moving object area and as about“0.3” to “0.5” to the background area may produce the result as shown inFIG. 19.

FIG. 20 is a block diagram showing a fourth example of the imageprocessing circuit 23 of the imaging apparatus according to theembodiment of the present invention. The image processing circuit 23according to the fourth example has a locus pixel frame selection unit221, a background pixel frame selection unit 222, a backgroundgeneration unit 231, a background image holding unit 261, thedegree-of-difference generation unit 240, the locus synthesis unit 250,the locus-synthesized image holding unit 270, and the locus-synthesizedimage display unit 290. The fourth embodiment is the same as the thirdexample in that the in-moving state of the moving object is to bedisplayed as more than one locus-synthesized image. However, while thethird embodiment is provided on the assumption that the frames from theinput images 111 are provided without being settled in advance, thefourth example assumes that the n frames of input images 110 arepredefined.

The background pixel frame selection unit 222 is to select the requiredpixel value for the generation of the background value out of the npieces of pixel values in the corresponding pixel positions as for the nframes of input images 110 (I1 to In) provided by being imaged in timeseries. While the third example has made the selection of all the nframes of input images to generate the background value, the fourthexample allows for the simplification of processing for generating thebackground value by using the thinned-out images provided from the nframes of input images. For instance, the thinning-out of the inputimages into “n/10” on the pixel values selected by the background pixelframe selection unit 222 enables the increase in processing speed.

The background generation unit 231 is to generate the background valuebased on the pixel value supplied from the background pixel frameselection unit 222, and is implemented with the same configuration as inthe case of the first example.

The background image holding unit 261 is to hold the background imagegenerated by the background generation unit 231. While the third examplehas made the sequential updating of the background image held in thebackground image holding unit 260 with the background generation unit230, the fourth example specifies that the background image oncegenerated is not subject to change, unless otherwise required.

The locus pixel frame selection unit 221 is to select the n pieces ofpixel values in the corresponding pixel positions as for the n frames ofinput images 110 (I1 to In) provided by being imaged in time series.Specifically, unlike the background pixel frame selection unit 222, thelocus pixel frame selection unit 221 makes the selection of all theinput images without the thinning-out.

It is to be noted that other configuration of the fourth example is thesame as that of the third example, and hence, its re-description isomitted.

In this manner, the fourth example provides, after calculating thebackground image frame only once, the locus synthesis sequentially inframe units using the calculated background image frame, permitting aprocessing cost to be lessened.

The operations of the imaging apparatus according to one embodiment ofthe present invention are now described with reference to the drawings.

FIG. 21 is a flowchart showing one procedure of the processing with thefirst and the second examples of the image processing circuit 23 of theimaging apparatus according to the embodiment of the present invention.

First, the pixel (or the frame) as the target for background generationis selected (Step S911). While this processing is performed with thepixel selection unit 120 in the first example, the second exampleemploys the background pixel selection unit 122 to perform thisprocessing. Then, the background image is generated from the selectedpixel (or selected frame) (Step S912). This processing is performed withthe background generation unit 130.

Subsequently, the pixel (or the frame) as the target for locus synthesisis selected (Step S913). While this processing is performed with thepixel selection unit 120 in the first example, the second embodimentemploys the locus pixel selection unit 121 to perform this processing.Then, the degree-of-difference between the selected pixel (or selectedframe) and the background is generated based on the selected pixel andthe background (Step S914). This processing is performed with thedegree-of-difference generation unit 140.

Then, the locus-synthesized image is outputted based on thedegree-of-difference (Step S915). This processing is performed with thelocus synthesis unit 150.

FIG. 22 is a flowchart showing one procedure of the processing with thethird example of the image processing circuit 23 of the imagingapparatus according to the embodiment of the present invention.

First, the frame as the target for locus synthesis is selected as theinput image (Step S931). This processing is performed with the frameselection unit 220. Thereafter, background image is updated with theselected frame (Step S932). This processing is performed with thebackground generation unit 230.

Subsequently, the degree-of-difference between the background image andthe input image is generated based on the background image and the inputimage (Step S934). This processing is performed with thedegree-of-difference generation unit 240.

Subsequently, the locus synthesis image is updated based on thedegree-of-difference (Step S935). This processing is performed with thelocus synthesis unit 250.

The selection of all the frames as the targets for locus synthesis leadsto termination of the processing, while the processing following theStep S931 is repeated in other cases (Step S936). The selection of theframes in the time series order in the Step S931 allows the locussynthesis to be provided in the time series order in the Step S935,followed by output of the locus synthesis result, permitting a locusmoving image to be played back.

FIG. 23 is a flowchart showing one procedure of the processing with thefourth example of the image processing circuit 23 of the imagingapparatus according to the embodiment of the present invention.

First, the frames as the targets for background generation are allselected (Step S941). This processing is performed with the backgroundpixel frame selection unit 222. Then, the background image is generatedwith the selected frames (Step S942). This processing is performed withthe background generation unit 231.

Subsequently, the frame as the target for locus synthesis is selected asthe input image (Step S943). This processing is performed with the locuspixel frame selection unit 221. Then, the degree-of-difference betweenthe background image and the input image is generated based on thebackground image and the input image (Step S944). This processing isperformed with the degree-of-difference generation unit 240.

Subsequently, the locus-synthesized image is updated based on thedegree-of-difference (Step S945). This processing is performed with thelocus synthesis unit 250.

The selection of all the frames as the targets for locus synthesis leadsto the termination of the processing, while the processing following theStep S943 is repeated in other cases (Step S946).

As described the above, according to one embodiment of the presentinvention, the still and/or moving image of the moving object locus maybe generated by, after generating the degree-of-difference between eachinput image pixel value and the background image pixel value, reflectingthe input image pixel values in the locus-synthesized image provided asthe output image depending on the generated degree-of-difference.

While the above embodiments of the present invention have been describedin relation to one configuration for generating the background imagewhen (or immediately before) generating the locus-synthesized image, itmay be appreciated that use of a preliminarily generated backgroundimage will do. For instance, like a modification shown in FIG. 24, thebackground image supplied to the degree-of-difference generation unit240 after being held in the background image holding unit 261 is to bestored in a background image storage unit 206. Afterwards, in generatingthe locus-synthesized image, the background image stored in thebackground image storage unit 206 is restored, causing the restoredbackground image to be held in the background image holding unit 261,without generation of the background image using the backgroundgeneration unit 231.

This processing may eliminate the need to generate the background imagefor several times of attempts to generate the locus-synthesized imagewith respect to the same input image. The calculation for the backgroundgeneration generally requires a higher cost, as compared with that forthe generation of the locus-synthesized image, so that no need togenerate the background image leads to significant effects. Forinstance, it is effective in cases such as in allowing the display unit43 to play back the locus moving image relating to scenes to be playedback after storage of more than one frame image provided by being takencontinuously, the locus-synthesized image and the background image.

The modification shown in FIG. 24 is the same as the fourth exampleshown in FIG. 20 in the configurations and operations of the locus pixelframe selection unit 221, the background pixel frame selection unit 222,the background generation unit 231, the background image holding unit261, the degree-of-difference generation unit 240, the locus synthesisunit 250 and the locus-synthesized image holding unit 270. Thismodification is provided on the assumption that, upon receipt of theinput image supplied from the A/D conversion circuit 22 through thesignal line 201, the image processing circuit 23 is to be operated tosupply the received input image, the background image and thelocus-synthesized image to the post-staged unit such as the D/Aconversion circuit 41 and the encoder/decoder 31 through the signallines 207, 208 and 209.

A camera signal processing unit 202 is to give, processing such as RGBsimultaneous processing, color matrix processing and gamma processing tothe input image provided by being converted into the digital signal bythe A/D conversion circuit 22, followed by outputting the resultantinput image. For normal image output, the resultant input image isoutputted from the camera signal processing unit 202 through the signalline 207 without being converted.

An input image holding unit 203 is to hold the input image to besupplied to the locus pixel frame selection unit 221 and the backgroundpixel frame selection unit 222. The n frames of input images (I1 to In)are herein supposed to be held, like the previously described fourthexample. An input image storage unit 205 is to store the input imagesupplied from the camera signal processing unit 202. The input imageholding unit 203 holds the supplied image from the camera signalprocessing unit 202 as the input image in a case where an image inprocess of being taken is used as the input image, or makes restorationof the stored image in the input image storage unit 205 as the inputimage before holding in a case where a past stored image is used as theinput image.

The background image storage unit 206 is to store the background imageheld in the background image holding unit 261. The background imagestorage unit 206 makes temporary storage of the background imagesupplied from the background image holding unit 261 through the signalline 208. Afterwards, the background image stored in the backgroundimage storage unit 206 is restored at need and then held in thebackground image holding unit 261. Specifically, in generating thebackground image at the time when (or immediately before) generating thelocus-synthesized image, the background image holding unit 261 holds thebackground image supplied from the background generation unit 231. Onthe other hand, in generating the locus-synthesized image using the paststored background image, the background image holding unit 261 makes therestoration of the background image stored in the background imagestorage unit 206 before holding.

The locus-synthesized image outputted from the locus synthesis unit 250is held in the locus-synthesized image holding unit 270, and is alsosupplied to the post-staged unit such as the D/A conversion circuit 41and the encoder/decoder 31 through the signal line 209.

FIG. 25 shows one configuration related to the operations of the imagingapparatus according to the embodiment of the present invention. It isherein assumed that the operation input acceptance unit 52 includes animage-taking operation acceptance unit 421 and an image-taking intervalsetting unit 422, the image processing circuit 23 includes an objectcondition detection unit 431, and the control unit 54 includes animage-taking start detection unit 441, an image-taking end detectionunit 442, an image-taking interval acquiring unit 443 and animage-taking control unit 444. These functions may be implemented usingother circuits etc.

The image-taking operation acceptance unit 421 is to accept animage-taking operation given by the user, and corresponds to a unit suchas the shutter button included in a digital still camera and a movingimage-taking button included in a digital video camera, for instance.The image-taking interval setting unit 422 is to permit a temporalinterval of each frame at the time of the image-taking operation to beset. The image-taking interval is to be used in a continuousimage-taking mode, and is preset by the user or settled fixedly. Theimage-taking interval acquiring unit 443 is to acquire the image-takinginterval provided by being set in the image-taking interval setting unit422.

The object condition detection unit 431 is to detect the condition ofthe object contained in the input image. The object condition detectionunit 431 generates the pixel-to-pixel difference on the temporallycontinuous input image frames, and then judges the object to be in astatic condition if there is no pixel-to-pixel difference, orconversely, in a moving condition if there is the pixel-to-pixeldifference.

The image-taking start detection unit 441 detects the occurrence of animage-taking start instruction-based on the image-taking operationaccepted in the image-taking operation acceptance unit 421. Theimage-taking end detection unit 442 detects the occurrence of animage-taking end instruction based on the image-taking operationaccepted in the image-taking operation. acceptance unit 421. As for anormal continuous image-taking mode, for instance, the occurrence of anaction of pushing down the shutter button involves the image-takingstart, while a release from the action of pushing down the shutterbutton involves the image-taking end.

Further, continuous image-taking control is also executable by givingattention to the condition of the object. For instance, it is allowableto give control so as to provide the image-taking end if the object isjudged to be in the static condition by the object condition detectionunit 431, with the shutter button pushed down.

The image-taking control unit 444 is to control the timing generator 51so as to provide a continuous image-taking operation at the image-takinginterval acquired by the image-taking interval acquiring unit 443,during a period from the detection of the image-taking start by theimage-taking start detection unit 441 to the detection of theimage-taking end by the image-taking end detection unit 442.

FIG. 26 is a flowchart showing one procedure of the processing wheneffecting the image-taking operation with the imaging apparatusaccording to one embodiment of the present invention. Theimage-taking-start is detected with the image-taking start detectionunit 441 (Step S951), followed by effecting of the image-takingoperation at the image-taking interval acquired by the image-takinginterval acquiring unit 443 (Step S952). This image-taking step isrepeated until the image-taking end is detected with the image-takingend detection unit 442 (Step S954). During this processing, thecondition of the object is detected with the object condition detectionunit 431 (Step S953).

In a case where the image-taking operation acceptance unit 421 providesacceptance of the release from the action of pushing down the shutterbutton, and/or the object condition detection unit 431 provides thedetection of the object to be in the static condition, the image-takingend is detected (Step S954).

Upon the end of the image-taking operation, the background image isgenerated in the background generation unit 231 (Step S955).Specifically, all the frames as the targets for background generationare selected as shown in FIG. 23 (Step S941), followed by a step (StepS942) of generating the background image with the selected frames.

Then, the locus-synthesized image is generated with the locus synthesisunit 250 (Step S956). Further, the input image and the background imageare stored in the input image storage unit 205 and the background imagestorage unit 206, respectively (Step S957).

FIG. 27 is a flowchart showing one procedure of the processing wheneffecting a playback operation with the imaging apparatus according tothe embodiment of the present, invention. A playback start instructiongiven through the operation input acceptance unit 52 is detected (StepS961), followed by acquisition of a direction and a speed of playback(Step S962). The direction and the speed of playback are to be acceptedwith the operation input acceptance unit 52. For instance, the operationinput acceptance unit 52 may include a forward playback button, areverse playback button and a fast forward button etc. are assumed.

In a case where the background image is stored in the Step S957, thebackground image is restored from the background image storage unit 206and is then held in the background image holding unit 261 (Step S963).Further, after the restoration of the input image from the input imagestorage unit 205, the input image is held in the input image holdingunit 203, followed by acquisition of the input image from the inputimage holding unit 203 in the time series order in the playbackdirection acquired in the Step S962 (Step S964). This processing permitsthe locus-synthesized image to be generated in the locus synthesis unit250 (Step S965), followed by effecting of the playback operation at theplayback speed acquired in the Step S962 (Step S966). The processing ofthe Steps from S964 to S966 is repeated on all the taken images relatingto the target scenes (Step S967).

While the above embodiments of the present invention have been describedin relation to one mode for implementing the image processing relevantto the imaging apparatus with the circuits, the contents of the aboveimage processing may be implemented with software. Specifically, theprocedures of the processing having been described in the aboveembodiments of the present invention may be understood as a methodhaving a series of the above steps, or alternatively, as a programadapted for a computer to perform the series of the above steps or arecording medium used to store the program. Alternatively, the presentinvention is not limited to the image processing circuit integrated withthe imaging apparatus, and an implementation in the form of anindependent image processing apparatus may be also made. Further, thepresent invention also allows for integration into an image reproducingapparatus and/or a mobile apparatus.

While the above embodiments of the present invention have been describedas related to one instance to embody the present invention, andrespectively provide correspondences with specific matters of thepresent invention involved in claims as follows, the present inventionis not limited to the above correspondences, and various changes andmodifications may be made in the present invention without departingfrom the spirit and scope thereof.

Specifically, in a claim, background generation means,degree-of-difference generation means and output image generation meansrespectively correspond to the background generation unit 130, thedegree-of-difference generation unit 140 and the locus synthesis unit150, for instance.

Further, in a claim, background pixel value holding means,occurrence-of-frequency holding means and reference value setting meansrespectively correspond to the background value holding unit 138, thebackground frequency holding unit 137 and the reference value settingunit 133, for instance. Further, background candidate generation meanscorresponds to the background determination unit 134 and the backgroundvalue calculation unit 135, and update determination means correspondsto the update determination unit 136 and the background frequencycounter 1361, for instance.

Furthermore, in a claim, output image holding means, synthesis ratiogeneration means and synthesis value calculation means respectivelycorrespond to the locus synthesis value holding unit 153, the locussynthesis ratio generation unit 151 and the locus synthesis valuecalculation unit 152, for instance.

Furthermore, in a claim, output image holding means, frame numberholding means and update determination means respectively correspond tothe locus synthesis value holding unit 156, the latest frame numberholding unit 155 and the update determination unit 154, for instance.

Furthermore, in a claim, output image holding means,degree-of-difference holding means and update determination meansrespectively correspond to the locus synthesis value holding unit 156,the maximum degree-of-difference holding unit 158 and the updatedetermination unit 157, for instance.

Furthermore, in a claim, image selection means, background generationmeans, degree-of-difference generation means and output image generationmeans respectively correspond to-the background image selection unit122, the background generation unit 130, the degree-of-differencegeneration unit 140 and the locus synthesis unit 150, for instance.

Furthermore, in a claim, background image holding means, output imageholding means, background generation means, degree-of-differencegeneration means and output image synthesis means respectivelycorrespond to the background image holding unit 260, thelocus-synthesized image holding unit 270, the background generation unit230, the degree-of-difference generation unit 240 and the locussynthesis unit 250, for instance.

Furthermore, in a claim, image selection means, background generationmeans, background image holding means, output image holding means,degree-of-difference generation means and output image synthesis meansrespectively correspond to the background pixel frame selection unit222, the background generation unit 231, the background image holdingunit 261, the locus-synthesized image holding unit 270, thedegree-of-difference generation unit 240 and the locus synthesis unit250, for instance.

Furthermore, in a claim, a step of generating, on each of correspondingpixel positions of more than one input image, a pixel value involved ina predetermined distribution among pixel values of the input images as abackground pixel value in the above pixel position; a step ofgenerating, on each of the pixel positions, a degree-of-differenceindicating a degree to which each input image pixel value has adifference from the background pixel value; and a step of generating, oneach of the pixel positions, an output image pixel value by reflectingthe input image pixel values depending on the degree-of-differencerespectively correspond to the Steps S912, S914 and S915, for instance.

Furthermore, in a claim, a step of selecting a predetermined number ofinput images as selected images from more than one input image; a stepof generating, on each of corresponding pixel positions of the selectedimages, a pixel value involved in a predetermined distribution amongpixel values of the selected images as a background pixel value in theabove pixel position; a step of generating, on each of the pixelpositions, a degree-of-difference indicating a degree to which eachinput image pixel value has a difference from the background pixelvalue; and a step of generating, on each of the pixel positions, anoutput image pixel value by reflecting the input image pixel valuesdepending on the degree-of-difference respectively correspond to theSteps S911, S912, S914 and S915, for instance.

Furthermore, in a claim, background image holding means and output imageholding means respectively correspond to the background image holdingunit 260 and the locus-synthesized image holding unit 270, for instance.Further, in the claim, a step of updating the background image pixelvalue held in the background image holding means based on apredetermined ratio and with pixel values in corresponding pixelpositions of input images allowed to enter in time series; a step ofgenerating a degree-of-difference indicating a degree to which eachinput image pixel value has a difference from the pixel value in thecorresponding pixel position of the updated background image provided bythe background generation means; and a step of synthesizing, on theoutput image pixel value, the pixel values in the corresponding pixelpositions of the input images depending on the degree-of-differencerespectively correspond to the Steps S932, S934 and S935, for instance.

Furthermore, in a claim, background image holding means and output imageholding means respectively correspond to the background image holdingunit 261 and the locus-synthesized image holding unit 270, for instance.Further, in the claim, a step of selecting a predetermined number ofinput images as selected images from more than one input image and astep of generating, on each of corresponding pixel positions of theselected images, a pixel value involved in a predetermined distributionamong pixel values of the selected images as a pixel value in acorresponding pixel position of a background image, causing thegenerated pixel value to be held in the background image holding meansrespectively corresponds to the Steps S941 and S942, for instance.Further, in the claim, a degree-of-difference generation step and anoutput image synthesis step respectively correspond to the Steps S944and S945, for instance.

Furthermore, in a claim, image selection means, background generationmeans, background image holding means, output image holding means,degree-of-difference generation means, output image synthesis means andbackground image storage means respectively correspond to the backgroundpixel frame selection unit 222, the background generation unit 231, thebackground image holding unit 261, the locus-synthesized image holdingunit 270, the degree-of-difference generation unit 240, the locussynthesis unit 250 and the background image storage unit 206, forinstance.

Furthermore, in a claim, background image holding means, backgroundimage storage means and output image holding means respectivelycorrespond to the background image holding unit 261, the backgroundimage storage unit 206 and the locus-synthesized image holding unit 270,for instance. Further, in the claim, a step of selecting a predeterminednumber of input images as selected images from more than one inputimage; a step of generating, on each of corresponding pixel positions ofthe selected images, a pixel value involved in a predetermineddistribution among pixel values of the selected images as a pixel valuein a corresponding pixel position of a background image, causing thegenerated pixel value to be held in the background image holding means;a step of causing the background image held in the background imageholding means to be stored in the background image storage means; and astep of causing the background image stored in the background imagestorage means to be restored and held in the background image holdingmeans respectively correspond to the Steps S941, S942, S957 and S963 forinstance. Further, in the claim, a degree-of-difference generation stepand an output image synthesis step respectively equal to the Steps S944and S945, for instance.

CROSS REFERENCES TO RELATED APPLICATIONS

The present document contains subject matter related to Japanese PatentApplications JP 2006-129092 and JP 2006-193435 filed in the JapanesePatent Office on May 8, 2006 and Jul. 13, 2006, respectively, the entirecontents of which being incorporated herein by reference.

1. An image processing apparatus comprising: background generation meansfor generating, on each of corresponding pixel positions of more thanone input image, a pixel value involved in a predetermined distributionamong pixel values of the input images as a background pixel value inthe pixel position; degree-of-difference generation means forgenerating, on each of the pixel positions, a degree-of-differenceindicating a degree to which each input image pixel value has adifference from the background pixel value; and output image generationmeans for generating, on each of the pixel positions, an output image;pixel value by reflecting the input image pixel values depending on thedegree-of-difference.
 2. The image processing apparatus according toclaim 1, wherein the background generation means generates a mostfrequently occurring pixel value as the pixel value involved in thepredetermined distribution.
 3. The image processing apparatus accordingto claim 2, wherein the background generation means includes: backgroundpixel value holding means for holding the background pixel value;frequency-of-occurrence holding means for holding afrequency-of-occurrence of the background pixel value; reference valuesetting means for setting a reference value for the input image pixelvalues; background candidate generation means for determining, as abackground candidate value, the input image pixel value that falls in apredetermined range of difference from the reference value; and updatedetermination means for, after counting the frequency-of-occurrence ofthe background candidate value, making updating so as to cause thefrequency-of-occurrence of the background candidate value to be held inthe frequency-of-occurrence holding means, and also, the backgroundcandidate value to be held as a latest background pixel value in thebackground pixel value holding means in a case where thefrequency-of-occurrence of the background candidate value is greaterthan the frequency-of-occurrence held in the frequency-of-occurrenceholding means.
 4. The image processing apparatus according to claim 1,wherein the output image generation means includes: output image holdingmeans for holding the output image pixel value; synthesis ratiogeneration means for generating, on each of the pixel positions, asynthesis ratio depending on the degree-of-difference; and synthesisvalue calculation means for synthesizing, on each of the pixelpositions, the input image pixel values with the output image pixelvalue held in the output image holding means depending on the synthesisratio, causing a synthesis result to be held as a latest output imagepixel value in the output image holding means.
 5. The image processingapparatus according to claim 1, wherein the output image generationmeans includes: output image holding means for holding the output imagepixel value; frame number holding means for holding a frame number ofthe output image; and update determination means for causing an inputimage frame number to be held in the frame number holding means, andalso, the input image pixel value to be held as a latest output imagepixel value in the output image holding means in a case where the framenumber of the input image is more up-to-date in time series than the.frame number held in the frame number holding means, provided that thedegree-of-difference given is not less than a predetermined degree. 6.The image processing apparatus according to claim 1, wherein the outputimage generation means includes: output image holding means for holdingthe output image pixel value; degree-of-difference holding means forholding the degree-of-difference on said output image; and updatedetermination means for causing the degree-of-difference on the inputimage to be held in the degree-of-difference holding means, and also,the input image pixel value to be held as a latest output image pixelvalue in the output image holding means in a case where thedegree-of-difference on the input image is greater than thedegree-of-difference held in the degree-of-difference holding means. 7.An image processing apparatus comprising: image selection means forselecting a predetermined number of input images as selected images frommore than one input image; background generation means for generating,on each of corresponding pixel positions of the selected images, a pixelvalue involved in a predetermined distribution among pixel values of theselected images as a background pixel value in the pixel position;degree-of-difference generation means for generating, on each of thepixel positions, a degree-of-difference indicating a degree to whicheach input image pixel value has a difference from the background pixelvalue; and output image generation means for generating, on each of thepixel positions, an output image pixel value by reflecting the inputimage pixel values depending on the degree-of-difference.
 8. An imageprocessing apparatus comprising: background image holding means forholding a background image pixel value; output image holding means forholding an output image pixel value; background generation means forupdating the background image pixel value held in the background imageholding means based on a predetermined ratio and with pixel values incorresponding pixel positions of input images allowed to enter in timeseries; degree-of-difference generation means for generating adegree-of-difference indicating a degree to which each input image pixelvalue has a difference from the pixel value in the corresponding pixelposition of the updated background image provided by the backgroundgeneration means; and output image synthesis means for synthesizing, onthe output image pixel value, the pixel values in the correspondingpixel positions of the input images depending on thedegree-of-difference, causing a synthesis result to be held as a latestoutput image pixel value in the output image holding means.
 9. An imageprocessing apparatus comprising: image selection means for selecting apredetermined number of input images as selected images from more thanone input image; background generation means for generating, on each ofcorresponding pixel positions of the selected images, a pixel valueinvolved in a predetermined distribution among pixel values of theselected images as a pixel value in a corresponding pixel position of abackground image; background image holding means for holding thebackground image pixel value; output image holding means for holding anoutput image pixel value; degree-of-difference generation means forgenerating a degree-of-difference indicating a degree to which eachinput image pixel value has a difference from the corresponding pixelvalue of the background image held in the background image holdingmeans; and output image synthesis means for synthesizing, on the outputimage pixel value, the pixel values in the corresponding pixel positionsof the input images depending on the degree-of-difference, causing asynthesis result to be held as a latest output image pixel value in theoutput image holding means.
 10. An image processing method comprising: astep of generating, on each of corresponding pixel positions of morethan one input image, a pixel value involved in a predetermineddistribution among pixel values of the input images as a backgroundpixel value in the pixel position; a step of generating, on each of thepixel positions, a degree-of-difference indicating a degree to whicheach input image pixel value has a difference from the background pixelvalue; and a step of generating, on each of the pixel positions, anoutput image pixel value by reflecting the input image pixel valuesdepending on the degree-of-difference.
 11. An image processing methodcomprising: a step of selecting a predetermined number of input imagesas selected images from more than one input image; a step of generating,on each of corresponding pixel positions of the selected images, a pixelvalue involved in a predetermined distribution among pixel values of theselected images as a background pixel value in the pixel position; astep of generating, on each of the pixel positions, adegree-of-difference indicating a degree to which each input image pixelvalue has a difference from the background pixel value; and a step ofgenerating, on each of the pixel positions, an output image pixel valueby reflecting the input image pixel values depending on thedegree-of-difference.
 12. An image processing method applied to an imageprocessing apparatus having background image holding means for holding abackground image pixel value and output image holding means for holdingan output image pixel value, the image processing method comprising: astep of updating the background image pixel value held in the backgroundimage holding means based on a predetermined ratio and with pixel valuesin corresponding pixel positions of input images allowed to enter intime series; a step of generating a degree-of-difference indicating adegree to which each input image pixel value has a difference from thepixel value in the corresponding pixel position of the updatedbackground image provided by the background generation means; and a stepof synthesizing, on the output image pixel value, the pixel values inthe corresponding pixel positions of said input images depending on saiddegree-of-difference, causing a synthesis result to be held as a latestoutput image pixel value in the output image holding means; wherein thesteps are repeated every time enter of the input images reoccurs.
 13. Animage processing method applied to an image processing apparatus havingbackground image holding means for holding a background image pixelvalue and output image holding means for holding an output image pixelvalue, the image processing method comprising: a step of selecting apredetermined number of input images as selected images from more thanone input image; a step of generating, on each of corresponding pixelpositions of the selected images, a pixel value involved in apredetermined distribution among pixel values of the selected images asa pixel value in a corresponding pixel position of the background image,causing the generated pixel value to be held in the background imageholding means; a degree-of-difference generation step of generating adegree-of-difference indicating a degree to which each input image pixelvalue has a difference from the corresponding pixel value of thebackground image held in the background image holding means; and anoutput image synthesis step of synthesizing, on the output image pixelvalue, the pixel values in the corresponding pixel positions of theinput images depending on the degree-of-difference, causing a synthesisresult to be held as a latest output image pixel value in the outputimage holding means; wherein the degree-of-difference generation stepand the output image synthesis step are repeated every time enter of theinput images reoccurs.
 14. A program adapted for causing a computer toperform processing including: a step of generating, on each ofcorresponding pixel positions of more than one input image, a pixelvalue involved in a predetermined distribution among pixel values of theinput images as a background pixel value in the pixel position; a stepof generating, on each of the pixel positions, a degree-of-differenceindicating a degree to which each input image pixel value has adifference from the background pixel value; and a step of generating, oneach of the pixel positions, an output image pixel value by reflectingthe input image pixel values depending on the degree-of-difference. 15.A program adapted for causing a computer to perform processingincluding: a step of selecting a predetermined number of input images asselected images from more than one input image; a step of generating, oneach of corresponding pixel positions of the selected images, a pixelvalue involved in a predetermined distribution among the pixel values ofthe selected images as a background pixel value in the pixel position; astep of generating, on each of the pixel positions, adegree-of-difference indicating a degree to which each input image pixelvalue has a difference from the background pixel value; and a step ofgenerating, on each of the pixel positions, an output image pixel valueby reflecting the input image pixel values depending on thedegree-of-difference.
 16. A program adapted for causing a computer toperform processing, applied to an image processing apparatus havingbackground image holding means for holding a background image pixelvalue and output image holding means for holding an output image pixelvalue, and including: a step of updating the background image pixelvalue held in the background image holding means based on apredetermined ratio and with pixel values in corresponding pixelpositions of input images allowed to enter in time series; a step ofgenerating a degree-of-difference indicating a degree to which eachinput image pixel value has a difference from the pixel value in thecorresponding pixel position of the updated background image provided bythe background generation means; and a step of synthesizing, on theoutput image pixel value, the pixel values in the corresponding pixelpositions of the input images depending on the degree-of-difference,causing a synthesis result to be held as a latest output image pixelvalue in the output image holding means; wherein the computer is adaptedto repeat the steps every time enter of the input images reoccurs.
 17. Aprogram adapted for causing a computer to perform processing, applied toan image processing apparatus having background image holding means forholding a background image pixel value and output image holding meansfor holding an output image pixel value, and including: a step ofselecting a predetermined number of input images as selected images frommore than one input image; a step of generating, on each ofcorresponding pixel positions of the selected images, a pixel valueinvolved in a predetermined distribution among pixel values of saidselected images as a pixel value in a corresponding pixel position ofthe background image, causing the generated pixel value to be held inthe background image holding means; a degree-of-difference generationstep of generating a degree-of-difference indicating a degree to whicheach input image pixel value has a difference from the correspondingpixel value of the background image held in the background image holdingmeans; and an output image synthesis step of synthesizing, on the outputimage pixel value, the pixel values in the corresponding pixel positionsof the input images depending on the degree-of-difference, causing asynthesis result to be held as a latest output image pixel value in theoutput image holding means; wherein the computer is adapted to repeatthe degree-of-difference generation step and the output image synthesisstep every time enter of the input images reoccurs.
 18. An imageprocessing apparatus comprising: image selection means for selecting apredetermined number of input images as selected images from more thanone input image; background generation means for generating, on each ofcorresponding pixel positions of the selected images, a pixel valueinvolved in a predetermined distribution among pixel values of theselected images as a pixel value in a corresponding pixel position of abackground image; background image holding means for holding thebackground image pixel value; output image holding means for holding anoutput image pixel value; degree-of-difference generation means forgenerating a degree-of-difference indicating a degree to which eachinput image pixel value has a difference from the. corresponding pixelvalue of the background image held in the background image holdingmeans; output image synthesis means for synthesizing, on the outputimage pixel value, the pixel values in the corresponding pixel positionsof the input images depending on the degree-of-difference, causing asynthesis result to be held as a latest output image pixel value in theoutput image holding means; and background image storage means forstoring the background image pixel value held in the background imageholding means; wherein the background image holding means makesrestoration of the background image stored in the background imagestorage means before holding, in a case where a required backgroundimage is contained in the background image storage means.
 19. An imageprocessing method applied to an image processing apparatus havingbackground image holding means for holding a background image pixelvalue, background image storage means for storing the background imageheld in the background image holding means, and output image holdingmeans for holding an output image pixel value, the image processingmethod comprising: a step of selecting a predetermined number of inputimages as selected images from more than one input image; a step ofgenerating, on each of corresponding pixel positions of the selectedimages, a pixel value involved in a predetermined distribution amongpixel values of the selected images as a pixel value in a correspondingpixel position of the background image, causing the generated pixelvalue to be held in the background image holding means; a step ofcausing the background image held in the background image holding meansto be stored in the background image storage means; a step of causingthe background image stored in the background image storage means to berestored and held in the background image holding means; adegree-of-difference generation step of generating adegree-of-difference indicating a degree to which each input image pixelvalue has a difference from the corresponding pixel value of thebackground image held in the background image holding means; and anoutput image synthesis step of synthesizing, on the output image pixelvalue, the pixel values in the corresponding pixel positions of theinput images depending on the degree-of-difference, causing a synthesisresult to be held as a latest output image pixel value in the outputimage holding means; wherein the degree-of-difference generation stepand the output image synthesis step are repeated every time enter of theinput images reoccurs.
 20. A program adapted for causing a computer toperform processing, applied to an image processing apparatus havingbackground image holding means for holding a background image pixelvalue, background image storage means for storing the background imageheld in the background image holding means, and output image holdingmeans for holding an output image pixel value, and including: a step ofselecting a predetermined number of input images as selected images frommore than one input image; a step of generating, on each ofcorresponding pixel positions of the selected images, a pixel valueinvolved in a predetermined distribution among pixel values of theselected images as a pixel value in a corresponding pixel position ofthe background-image, causing the generated pixel value to be held inthe background image holding means; a step of causing the backgroundimage held in the background image holding means to be stored in thebackground image storage means; a step of causing the background imagestored in the background image storage means to be restored and held inthe background image holding means; a degree-of-difference generationstep of generating a degree-of-difference indicating a degree to whicheach input image pixel value has a difference from the correspondingpixel value of the background image held in the background image holdingmeans; and an output image synthesis step of synthesizing, on the outputimage pixel value, the pixel values in the corresponding pixel positionsof the input images depending on the degree-of-difference, causing asynthesis result to be held as a latest output image pixel value in theoutput image holding means; wherein the computer is adapted to repeatthe degree-of-difference generation step and the output image synthesisstep every time enter of the input images reoccurs.